Transferring or migrating portions of data objects, such as block-level data migration or chunk-based data migration

ABSTRACT

A system and method for migrating data objects based on portions of the data objects is described. The system may transfer portions of files, folders, and other data objects from primary storage to secondary storage based on certain criteria, such as time-based criteria, age-based criteria, and so on. An increment may be one or more blocks of a data object, or one or more chunks of a data object, or other segments that combine to form or store a data object. For example, the system identifies one or more blocks of a data object that satisfy a certain criteria, and migrates the identified blocks. The system may determine that a certain number of blocks of a file have not been modified or called by a file system in a certain time period, and migrate these blocks to secondary storage.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a Continuation of U.S. patent application Ser. No.14/132,458 filed on Dec. 18, 2013, which is a divisional of U.S. patentapplication Ser. No. 12/553,199, filed Sep. 3, 2009, entitledTRANSFERRING OR MIGRATING PORTIONS OF DATA OBJECTS, SUCH AS BLOCK-LEVELDATA MIGRATION OR CHUNK-BASED DATA MIGRATION, which claims priority toU.S. Patent Application No. 61/096,587, filed on Sep. 12, 2008, entitledTRANSFERRING OR MIGRATING PORTIONS OF DATA OBJECTS, SUCH AS BLOCK-LEVELDATA MIGRATION OR CHUNK-BASED DATA MIGRATION, each of which isincorporated by reference in its entirety.

BACKGROUND

Data storage systems contain large amounts of data. This data includespersonal data, such as financial data, customer/client/patient contactdata, audio/visual data, and much more. Computer systems often containword processing documents, engineering diagrams, spreadsheets, businessstrategy presentations, email mailboxes, and so on. With theproliferation of computer systems and the ease of creating content, theamount of content in an organization has expanded rapidly. Even smalloffices often have more information stored than any single employee canknow about or locate.

To that end, both companies and individuals rely on data storage systemsto store, protect, and/or hold old data, such as data no longer activelyneeded. Often, these data storage systems perform data migration, movingdata from primary storage (containing actively needed data) to secondarystorage (such as backup storage or archives). Typical data storagesystems transfer data in the forms of files, folders, and so on. Forexample, the typical data storage system may transfer data from a datastore associated with a user to secondary storage while maintaining thestructure and application format of the files themselves.

To restore the data, these systems then require knowledge ofapplications that create the data. Additionally, some files, can be verylarge, and restoring a large file can be costly, time consuming, andresource intensive.

The need exists for a system that overcomes the above problems, as wellas one that provides additional benefits. Overall, the examples hereinof some prior or related systems and their associated limitations areintended to be illustrative and not exclusive. Other limitations ofexisting or prior systems will become apparent to those of skill in theart upon reading the following Detailed Description.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating components of a data streamutilized by a suitable data storage system.

FIG. 2 is a block diagram illustrating an example of a data storagesystem.

FIG. 3 is a block diagram illustrating an example of components of aserver used in data storage operations.

FIG. 4 is a block diagram illustrating a system for performingincrement-based data migration.

FIG. 5 is a block diagram illustrating the intermediate component ofFIG. 4.

FIGS. 6A and 6B are schematic diagrams illustrating a data store beforeand after a block-based data migration, respectively.

FIG. 7 is a flow diagram illustrating a routine for performingblock-level data migration.

FIG. 8 is a block diagram illustrating a system for providingchunk-based data migration and/or restoration.

FIG. 9 is a flow diagram illustrating a routine for performingchunk-level data migration.

FIG. 10 is flow diagram illustrating a routine for block-based orchunk-based data restoration and modification.

DETAILED DESCRIPTION Overview

Described in detail herein is a system and method that transfers ormigrates data objects (such as files, folders, data stores, and/ordiscrete data component(s) by migrating segments, portions, increments,or proper subsets of the data objects. The system may transferincrements of files, folders, and other data objects from primarystorage (or other sources) to secondary storage based on certaincriteria, such as time-based criteria, age-based criteria, and so on. Anincrement may be one or more blocks of a data object, or one or morechunks of a data object, or other portions that combine to form, store,and/or contain a data object, such as a file.

In some examples, the system performs block-based migration of data.That is, the system identifies one or more blocks of a data object thatsatisfy a certain criteria, and migrates the identified blocks. Forexample, the system may determine that a certain number of blocks of afile have not been modified or called by a file system within a certaintime period, and migrate these blocks to secondary storage. The systemthen maintains the other blocks of the file in primary storage. In somecases, the system automatically migrates data without requiring userinput. Additionally, the migration may be transparent to a user.

In some examples, the system performs chunk-based migration of data. Achunk is, for example, a group or set of blocks. One or more chunks maycomprise a file, folder, or other data object. The system identifies oneor more chunks of a data object that satisfy a certain criteria, andmigrates the identified chunks. For example, the system may determinethat a certain number of chunks of a file have not been modified orcalled by a file system in a certain time period, and migrate thesechunks to secondary storage. The system then maintains the other chunksof the file in primary storage. Further details regarding chunks andchunk-based storage may be found in U.S. Patent Application No.61/180,791, entitled BLOCK-LEVEL SINGLE INSTANCING, filed May 22, 2009.

In some examples, the system leverages the block-based or chunk-baseddata migration in order to restore portions of data objects withoutrestoring entire data objects. For example, the system can restore oneor more blocks of a file, present the data contained by the blocks,receive modifications to the data, and update the blocks, and hence thefile.

The system will now be described with respect to various examples. Thefollowing description provides specific details for a thoroughunderstanding of, and enabling description for, these examples of thesystem. However, one skilled in the art will understand that the systemmay be practiced without these details. In other instances, well-knownstructures and functions have not been shown or described in detail toavoid unnecessarily obscuring the description of the examples of thesystem.

The terminology used in the description presented below is intended tobe interpreted in its broadest reasonable manner, even though it isbeing used in conjunction with a detailed description of certainspecific examples of the system. Certain terms may even be emphasizedbelow; however, any terminology intended to be interpreted in anyrestricted manner will be overtly and specifically defined as such inthis Detailed Description section.

Suitable System

Referring to FIG. 1, a block diagram illustrating components of a datastream utilized by a suitable data storage and recovery system, such asa system that performs block-based and/or chunk-based data migration, isshown. The stream 110 may include a client 111, a media agent 112, and asecondary storage device 113. For example, in storage operations, thesystem may store, receive and/or prepare data, such as blocks or chunks,to be stored, copied or backed up at a server or client 111. The systemmay then transfer the data to be stored to media agent 112, which maythen refer to storage policies, schedule policies, and/retentionpolicies (and other policies) to choose a secondary storage device 113.The media agent 112 may include or be associated with an intermediatecomponent, to be discussed herein.

The secondary storage device 113 receives the data from the media agent112 and stores the data as a secondary copy, such as a backup copy.Secondary storage devices may be magnetic tapes, optical disks, USB andother similar media, disk and tape drives, and so on. Of course, thesystem may employ other configurations of stream components not shown inthe Figure.

Referring to FIG. 2, a block diagram illustrating an example of a datastorage and recovery system 200 is shown. Data storage systems maycontain some or all of the following components, depending on the needsof the system. FIG. 2 and the following discussion provide a brief,general description of a suitable computing environment in which thesystem can be implemented. Although not required, aspects of the systemare described in the general context of computer-executableinstructions, such as routines executed by a general-purpose computer,e.g., a server computer, wireless device or personal computer. Thoseskilled in the relevant art will appreciate that the system can bepracticed with other communications, data processing, or computer systemconfigurations, including: Internet appliances, network PCs,mini-computers, mainframe computers, and the like. Indeed, the terms“computer,” “host,” and “host computer” are generally usedinterchangeably herein, and refer to any of the above devices andsystems, as well as any data processor.

Aspects of the system can be embodied in a special purpose computer ordata processor that is specifically programmed, configured, orconstructed to perform one or more of the computer-executableinstructions explained in detail herein. Aspects of the system can alsobe practiced in distributed computing environments where tasks ormodules are performed by remote processing devices, which are linkedthrough a communications network, such as a Local Area Network (LAN),Wide Area Network (WAN), Storage Area Network (SAN), Fibre Channel, orthe Internet. In a distributed computing environment, program modulesmay be located in both local and remote memory storage devices.

Aspects of the system may be stored or distributed on computer-readablemedia, including tangible storage media, such as magnetically oroptically readable computer discs, hard-wired or preprogrammed chips(e.g., EEPROM semiconductor chips), nanotechnology memory, biologicalmemory, or other data storage media. Indeed, computer implementedinstructions, data structures, screen displays, and other data underaspects of the system may be distributed over the Internet or over othernetworks (including wireless networks), on a propagated signal on apropagation medium (e.g., an electromagnetic wave(s), a sound wave,etc.) over a period of time, or they may be provided on any analog ordigital network (packet switched, circuit switched, or other scheme).Those skilled in the relevant art will recognize that portions of thesystem reside on a server computer, while corresponding portions resideon a client computer, and thus, while certain hardware platforms aredescribed herein, aspects of the system are equally applicable to nodeson a network.

For example, the data storage system 200 contains a storage manager 210,one or more clients 111, one or more media agents 112, and one or morestorage devices 113. Storage manager 210 controls media agents 112,which may be responsible for transferring data to storage devices 113.Storage manager 210 includes a jobs agent 211, a management agent 212, adatabase 213, and/or an interface module 214. Storage manager 210communicates with client(s) 111. One or more clients 111 may access datato be stored by the system from database 222 via a data agent 221. Thesystem uses media agents 112, which contain databases 231, to transferand store data into storage devices 113. Client databases 222 maycontain data files and other information, while media agent databasesmay contain indices and other data structures that store the data atsecondary storage devices, for example.

The data storage and recovery system may include software and/orhardware components and modules used in data storage operations. Thecomponents may be storage resources that function to copy data duringstorage operations. The components may perform other storage operations(or storage management operations) other that operations used in datastores. For example, some resources may create, store, retrieve, and/ormigrate primary or secondary data copies of data. Additionally, someresources may create indices and other tables relied upon by the datastorage system and other data recovery systems. The secondary copies mayinclude snapshot copies and associated indices, but may also includeother backup copies such as HSM copies, archive copies, auxiliarycopies, and so on. The resources may also perform storage managementfunctions that may communicate information to higher level components,such as global management resources.

In some examples, the system performs storage operations based onstorage policies, as mentioned above. For example, a storage policyincludes a set of preferences or other criteria to be considered duringstorage operations. The storage policy may determine or define a storagelocation and/or set of preferences about how the system transfers datato the location and what processes the system performs on the databefore, during, or after the data transfer. In some cases, a storagepolicy may define a logical bucket in which to transfer, store or copydata from a source to a data store, such as storage media. Storagepolicies may be stored in storage manager 210, or may be stored in otherresources, such as a global manager, a media agent, and so on. Furtherdetails regarding storage management and resources for storagemanagement will now be discussed.

Referring to FIG. 3, a block diagram illustrating an example ofcomponents of a server used in data storage operations is shown. Aserver, such as storage manager 210, may communicate with clients 111 todetermine data to be copied to storage media. As described above, thestorage manager 210 may contain a jobs agent 211, a management agent212, a database 213, and/or an interface module. Jobs agent 211 maymanage and control the scheduling of jobs (such as copying data files)from clients 111 to media agents 112. Management agent 212 may controlthe overall functionality and processes of the data storage system, ormay communicate with global managers. Database 213 or another datastructure may store storage policies, schedule policies, retentionpolicies, or other information, such as historical storage statistics,storage trend statistics, and so on. Interface module 215 may interactwith a user interface, enabling the system to present information toadministrators and receive feedback or other input from theadministrators or with other components of the system (such as viaAPIs).

In some examples, the system performs some or all the operationsdescribed herein using an intermediate component, virtual storagedevice, virtual device driver, virtual disk driver, or otherintermediary capable of mounting to a file system and communicating witha storage device. That is, an intermediate component may communicativelyreside between a file system and a primary data store that contains datacreated by the file system and a secondary data store. The intermediatecomponent enables flexibility during data restoration, enabling a filesystem to indirectly access a secondary copy of data in order toidentify information associated with data stored by the secondary copy,among other benefits.

Data Migration System

Referring to FIG. 4, a block diagram illustrating a system forperforming portion-based data migration is shown. The system componentsinclude a data creation and/or modification component 410, anintermediate component 420, and a data storage component 430. Therestore component 410 may include a client portion 415, such as a clientportion that receives input from users. A file system 417, as discussedherein, may organize and provide data to applications, user interfaces,and so on to the user, among other things. The file system creates,updates, modifies, and/or removes data from a data store, based on inputfrom users. The file system 417 may store the created data in one ormore data stores, such as a local database 418 that provides primarystorage. For example, the database 418 may be a hard drive or hard diskthat stores data produced by the file system as primary copies orproduction copies of the data. The system components may also include anintermediate component 420 (further described herein), such as a virtualdisk driver. The intermediate component 420 communicates with a diskdriver 435 and mounted disk 437, which together may act as the datastorage component 430. Additionally, the intermediate component 420 maybe located between the file system 417 and database 418. The datastorage component provides secondary storage, and may store secondarycopies of data generated by the file system 417, such as secondarycopies of primary copies stored in database 418.

Referring to FIG. 5, a block diagram illustrating the intermediatecomponent 420 of FIG. 4 is shown. The intermediate component 420includes a restore module 510 that may contain its own file system 515.The restore module 510 (or component, sub-system, and so on), maycommunicate with a file system, such as the file system 417. Furtherdetails with respect the functionality of the restore module 510 isdescribed herein.

The intermediate component 420 may also include a storage device module520 that communicates with storage devices, such as disk driver 435 anddisk 437 (or other fixed or removable media). The storage device module520 may include an index 525 or allocation table that identifiesavailable media for data storage, contains information associated withdata stored via the intermediate component 420, and so on.

The intermediate component 420 may also include a cache 530 (or, a cachemodule or interface that communicates with an external cache), and/orother agents or modules 540, such as modules that index files, classifyfiles, manage files or information, and so on.

Block-Based Data Migration

Block-level migration, or block-based data migration, involves migratingdisk blocks from a primary data store (e.g., a disk partition) tosecondary media. Using block-level migration, a data storage systemtransfers blocks on a disk partition that have not been recentlyaccessed to secondary storage, freeing up space on the disk. In order toexpand the database, the system moves data from the database to otherlocations, such as other databases or storage locations. Typically, suchexpansion requires knowledge of the database, such as the databaseapplication, the database schema, and so on. However, using block-levelmigration, the system can expand or extend a database without anyknowledge of the applications or schema of the database, providing fortransparent migration and/or restoration of data from one storagelocation to another. This can be helpful when migrating data fromvirtual machines that contain large files, (e.g., large files created byapplications such as Vmware, Microsoft Virtual Server, and so on). Thesystem may implement block-level migration processes as software devicedrivers, but may also implement block-level migration in disk hardware.

As described herein, the system can transfer or migrate certain blocksof a data object from one data store to another, such as from primarystorage that contains a primary copy of the data object to secondarystorage that contains or will contain a secondary copy of the primarycopy of the data object. Referring to FIGS. 6A-6B, a schematic diagramillustrating contents of two data stores before and after a block-baseddata migration is shown. In FIG. 6A, a first data store 610 containsprimary copies (i.e., production copies) of two data objects, a firstdata object 620 and a second data object 630. The first data objectcomprises blocks A and A¹, where blocks A are blocks that satisfy ormeet certain storage criteria (such as blocks that have not beenmodified since creation or not been modified within a certain period oftime) and blocks A′ are blocks that do not meet the criteria (such asblocks that have been modified within the certain time period). Thesecond data object comprises blocks B and B′, where blocks B satisfy thecriteria and blocks B′ do not meet the criteria.

FIG. 6B depicts the first data store 610 after a block-based datamigration of the two data objects 620 and 630. In this example, thesystem only transfers the data from blocks that satisfy a criteria(blocks A and B) from the first data store 610 to a second data store640, such as secondary storage 642, 644. The secondary storage mayinclude one or more magnetic tapes, one or more optical disks, and soon. The system maintains data in the remaining blocks (blocks A′ and B′)within the first data store 610.

The system can perform file system data migration at a block level,unlike previous systems that only migrate data at the file level (thatis, they have a file-level granularity). By tracking migrated blocks,the system can also restore data at the block level, which may avoidcost and time problems associated with restoring data at the file levelor may assist in defragmenting a storage device. Further detailsregarding the block-level restoration of data is be discussed herein.

Referring to FIG. 7, a flow diagram illustrating a routine 700 forperforming block-level data migration is shown. In step 710, the systemidentifies data blocks within a data store that satisfy a certaincriteria. The system may track data blocks and access the blocks viaAPIs. The data store may be a database associated with a file system, aSQL database, a Microsoft Exchange mailbox, and so on. The system maycompare some or all of the blocks (or, information associated with theblocks) of the data store with predetermined criteria. The predeterminedcriteria may be time-based criteria within a storage policy or dataretention policy.

In some examples, the system identifies blocks set to be “aged off” fromthe data store. That is, the system identifies blocks created, changed,or last modified before a certain date and time. For example, the systemmay review a data store for all data blocks that satisfy a criterion orcriteria. The data store may be an electronic mailbox or personalfolders (.pst) file for a Microsoft Exchange user, and the criterion maydefine, for example, all blocks or emails last modified or changedthirty days ago or earlier. The system compares information associatedwith the blocks, such as metadata associated with the blocks, to thecriteria, and identifies all blocks that satisfy the criteria. Forexample, the system identifies all blocks in the .pst file not modifiedwithin the past thirty days. The identified blocks may include all theblocks for some emails and/or a portion of the blocks for other emails.That is, for a given email (or data object), a first portion of theblocks that include the email may satisfy the criteria, while a secondportion of the blocks that include the same email may not satisfy thecriteria. In other words, a file or a data object can be divided intoparts or portions, and only some of the parts or portions change.

To determine which blocks have changed, and when, the system can monitorthe activity of the file system via the intermediate component 420,(e.g., the virtual device driver). The system may store a datastructure, such as a bitmap, table, log, and so on within the cache 530or other memory of the intermediate component 420, and update the bitmapwhenever the file system calls the database 418 to access and update orchange data blocks within the database 418. The intermediate component420 traps the command to the disk driver, where that command identifiescertain blocks on a disk for access or modifications, and writes to thebitmap the changed blocks and the time of the change. The bitmap mayinclude information such as an identification of changed blocks and adate and a time the blocks were changed. The bitmap, which may be atable, data structure, or group of pointers, such as a snapshot, mayalso include other information, such as information that maps file namesto blocks, information that maps chunks to blocks and/or file names, andso on. Table 1 provides entry information for a bitmap tracking theactivity of a file system with the “/users” directory:

TABLE 1 Blocks Date and Time Modified /users/blocks1-100 Sep. 8, 2008@14:30 /users/blocks101-105 Sep. 4, 2008 @12:23 /users2/blocks106-110Sep. 4, 2008 @11:34 /users3/blocks110-1000 Aug. 5, 2008 @10:34

Thus, if a storage policy identified the time 08.30.2008 @ 12:00 as athreshold time criteria, where data modified after the time is to beretained, the system would identify, in step 710, blocks 110-1000 ashaving satisfied the criteria. Thus, the system, via the intermediatecomponent 420, can monitor what blocks are requested by a file system,and act accordingly, as described herein.

In step 720, the system transfers data within the identified blocks fromthe data store to a media agent, to be stored in a different data store.The system may perform some or all of the processes described withrespect to FIGS. 1-3 when transferring the data to the media agent. Forexample, before transferring data, the system may review a storagepolicy as described herein to select a media agent, such as media agent112, based on instructions within the storage policy. In step 725, thesystem optionally updates an allocation table, such as a file allocationtable (FAT) for a file system associated with the data store, toindicate the data blocks that no longer contain data and are now free toreceive and store data from the file system.

In step 730, via the media agent, the system stores data from the blocksto a different data store. In some cases, the system, via the mediaagent, stores the data from the blocks to a secondary storage device,such as a magnetic tape or optical disk. For example, the system maystore the data from the blocks in secondary copies of the data store,such as a backup copy, an archive copy, and so on. In some cases, thesystem stores the data from the blocks to a storage device located nearand/or associated with the data store, such as to a quick recoveryvolume that facilitates quick restores of data.

The system may create, generate, update, and/or include an allocationtable, (such as a table for the data store) that tracks the transferreddata and the data that was not transferred. The table may includeinformation identifying the original data blocks for the data, the nameof the data object, the location of any transferred data blocks, and soon. For example, Table 2 provides entry information for an example .pstfile:

TABLE 2 Name of Data Object Location of data Email1 C:/users/blocks1-100Email2.1 (body of email) C:/users/blocks101-120 Email2.2 (attachment)X:/remov1/blocks1-250 Email3 X:/remov2/blocks300-500

In the above example, the data for “Email2” is stored in two locations,a local data store (C:/) and an off-site data store (X:/). The systemmaintains the body of the email, recently modified or accessed, at alocation within a data store associated with a file system,“C:/users/blocks101-120.” The system stores the attachment, not recentlymodified or accessed, in a separate data store, “X:/remov1/blocks1-250.”Of course, the table may include other information, fields, or entriesnot shown. For example, when the system stored data to tape, the tablemay include tape identification information, tape offset information,and so on.

Chunk-Based Data Migration

Chunked file migration, or chunk-based data migration, involvessplitting a data object into two or more portions of the data object,creating an index that tracks the portions, and storing the data objectto secondary storage via the two or more portions. Among other things,the chunk-based migration provides for fast and efficient storage of adata object. Additionally, chunk-based migration facilitates fast andefficient recall of a data object, such as the large files describedherein. For example, if a user modifies a migrated file, chunk-basedmigration enables a data restore component to only retrieve from, andmigrate back to, secondary storage the chunk containing the modifiedportion of the file, and not the entire file. In some cases, chunk-basedmigration may collaborate with components that provide file formatand/or database schema information in order to facilitate data recovery.

As described above, in some examples the system migrates chunks of data(sets of blocks) that comprise a data object from one data store toanother. Referring to FIG. 8, a block diagram illustrating a system 800for providing chunk-based data migration and/or restoration is shown.The system 800 includes a file system 810, a callback layer 820, whichinteracts with the file system, and a device driver 830, which readsfrom and writes data to a data store 840 such as removable mediaincluding magnetic tapes, optical disks, and so on. Further details withrespect to the callback layer 820 will be described herein.

As described above, the system migrates data via one or more chunks,such as sets of blocks. A data object, such as a file, may comprise twoor more chunks. A chunk may be a logical division of a data object. Forexample, a .pst file may include two or more chucks: a first chunk thatstores data associated with an index of a user's mailbox, and one ormore chunks that stores email, attachments, and so on within the user'smailbox. A chunk is a proper subset of all the blocks comprising a file.That is, for a file consisting of n blocks, the largest chunk of thefile comprises at most n−1 blocks.

The system 800 may include a chunking component 815 that divides dataobjects, such as files, into chunks. The chunking component 815 mayreceive files to be stored in database 418, divide the files into two ormore chunks, and store the files as two or more chunks in database 418.The chunking component 815 may update an index that associatedinformation associated with files with the chunks of the file, the datablocks of the chunks, and so on.

The chunking component 815 may perform different processes whendetermining how to divide a data object. For example, the chunkingcomponent 815 may include indexing, header, and other identifyinginformation or metadata in a first chunk, and include the payload inother chunks. The chunking component 815 may follow a rules-basedprocess when dividing a data object. The rules may define a minimum ormaximum data size for a chunk, a time of creation for data within achunk, a type of data within a chunk, and so on.

For example, the chunking component 815 may divide a user mailbox (suchas a .pst file) into a number of chunks, based on various rules thatassign emails within the mailbox to chunks based on the metadataassociated with the emails. The chunking component 815 may place anindex of the mailbox in a first chunk and the emails in other chunks.The chunking component 815 may then divide the other chunks based ondates of creation, deletion or reception of the emails, size of theemails, sender of the emails, type of emails, and so on. Thus, as anexample, the chunking component may divide a mailbox as follows:

User1/Chunk1 Index User1/Chunk2 Sent emails User1/Chunk3 Received emailsUser1/Chunk4 Deleted emails User1/Chunk5 All Attachments.

Of course, other divisions are possible. Chunks may not necessarily fallwithin logical divisions. For example, the chunking component may dividea data object based on information or instructions not associated withthe data object, such as information about data storage resources,information about a target secondary storage device, historicalinformation about previous divisions, and so on.

The system may perform chunking at various times or in differentlocations of a data storage system. For example, although FIG. 8 showsthe chunking component 815 at file system 810, the system may locate thechunking component at the device driver 830, at an intermediatecomponent, or other locations. In some cases, the system may utilize thechunking component 815 to divide data already in secondary storage intochunks. For example, a data storage system may retrieve data objectsunder management that were transferred to secondary storage usingfile-based data migration, divide the data objects into two or morechunks, and migrate the data objects based to storage using thechunk-based data migration discussed herein. Thus, future restoration ofthe data objects may be faster and easier because the data objects aredivided into chunks.

Referring to FIG. 9, a flow diagram illustrating a routine 900 forperforming chunk-level data migration is shown. In step 910, the systemidentifies chunks of data blocks within a data store that satisfy one ormore criteria. The data store may store large files (>50 MB), such asdatabases associated with a file system, SQL databases, MicrosoftExchange mailboxes, virtual machine files, and so on. The system maycompare some or all of the chunks (or, information associated with thechunks) of the data store with predetermined and/or dynamic criteria.The predetermined criteria may be time-based criteria within a storagepolicy or data retention policy. The system may review an index with thechunking component 815 when comparing the chunks with applicablecriteria.

In step 920, the system transfers data within the identified chunks fromthe data store to a media agent, to be stored in a different data store.The system may perform some or all of the processes described withrespect to FIGS. 1-3 when transferring the data to the media agent. Forexample, the system may review a storage policy assigned to the datastore and select a media agent based on instructions within the storagepolicy. In step 925, the system optionally updates an allocation table,such as a file allocation table (FAT) for a file system associated withthe data store, to indicate the data blocks that no longer contain dataand are now free to receive and store data from the file system.

In some examples, the system monitors the transfer of data from the filesystem to the data store via the callback layer 820. The callback layer820 may be a layer, or additional file system, that resides on top ofthe file system 810. The intermediate layer 820 may intercept datarequests from the file system 810, in order to identify, track and/ormonitor the chunks requested by the file system 810 and storeinformation associated with these requests in a data structure, such asa bitmap similar to the one shown in Table 1. Thus, the intermediatelayer 820 stores information identifying when chunks are accessed bytracking calls from the file system 810 to the data store 840. Forexample, Table 3 provides entry information for a bitmap tracking callsto a data store:

TABLE 3 Chunk of File1 Access Time File1.1 Sep. 5, 2008 @12:00 File1.2Sep. 5, 2008 @12:30 File1.3 Sep. 5, 2008 @13:30 File1.4 Jun. 4, 2008 @12:30

In this example, the file system 810 creates a data object named“File1,” using the chunking component to divide the file into fourchunks: “File1.1,” “File1.2,” “File1.3,” and “File1.4.” The file system810 stores the four chunks to data store 840 on 06.04.2008. According tothe table, the file system has not accessed File1.4 since its creation,and most recently accessed the other chunks on Sep. 5, 2008. Of course,Table 3 may include other or different information, such as informationidentifying a location of the chunks, information identifying the typeof media storing the chunks, information identifying the blocks withinthe chunk, and/or other information or metadata.

In step 930, via the media agent, the system stores the data from thechunks to a different data store. In some cases, the system, via themedia agent, stores the data to a secondary storage device, such as amagnetic tape or optical disk. For example, the system may store thedata in secondary copies of the data store, such as a backup copy, andarchive copy, and so on. In some cases, the system stores the data to astorage device located near and/or associated with the data store, suchas to a quick recovery volume.

Data Recovery

The system, using the block-based or chunk-based data migrationprocesses described herein, is able to restore portions of files insteadof entire files, such as individual blocks or chunks that compriseportions of the files. Referring to FIG. 10, a flow diagram illustratinga routine 1000 for block-based or chunk-based data restoration andmodification is shown. In step 1010, the system, via a restore or datarecovery component, receives a request to modify a file located in adata store. For example, a user submits a request to a file system toprovide an old copy of a large Powerpoint presentation so the user canmodify a picture located on slide 5 of 300 of the presentation. Forexample, the data recovery component 410 works with the file system 417and the data store 430.

In step 1020, the system identifies one or more blocks or one or morechunks associated with the request. For example, the system looks to atable similar to Table 2, and identifies blocks associated with page 5of the presentation and blocks associated with an table of contents ofthe presentation.

In step 1030, the system retrieves the identified blocks or chunks andpresents them to the user. For example, the system only retrieves page 5and table of contents of the presentation and presents the pages to theuser.

In step 1040, the system, via the file system, modifies the retrievedblocks or chunks via the file system. For example, the user updates thePowerpoint presentation to include a different picture. In step 1050,the system transfers data associated with the modified blocks or chunksto the data store. For example, the system transfers the modified page 5to the data store. The system may also update a table that tracks accessto the data store, such as Table 1 or Table 3.

Thus, the system, leveraging block-based or chunk-based data migrationduring data storage, restores only portions of data objects required bya file system. Such restoration can be, among other benefits,advantageous over systems that perform file-based restoration, becausethose systems restore entire files, which can be expensive, timeconsuming, and so on. Some files, such as .pst files, may contain largeamounts of data. File-based restoration can therefore be inconvenientand cumbersome, among other things, especially when a user only requiresa small portion of a large file.

For example, a user submits a request to the system to retrieve an oldemail stored in a secondary copy on removable media. The systemidentifies a portion of a .pst file associated with the user thatcontains a list of old emails, and retrieves the list. That is, thesystem has knowledge of the chunk that includes the list (e.g., achunking component may always include the list in a first chunk of adata object), accesses the chunk, and retrieves the list. The otherportions (e.g., all the emails with the .pst file), are not retrievedfrom media. The user selects the desired email from the list. Thesystem, via an index that associates chunks with data (such as an indexsimilar to Table 2), identifies the chunk that contains the email, andretrieves the chunk for presentation to the user. The index may includeinformation about the chunks, information about the data objects (suchas file formats, database schemas, application specific information, andso on).

Thus, the system is able to restore the email without restoring theentire mailbox (.pst file) associated with the user. That is, althoughan entire data object is in storage, the system is able to retrieve aportion of the entire data object by leveraging the processes describedherein.

CONCLUSION

From the foregoing, it will be appreciated that specific examples of thedata recovery system have been described herein for purposes ofillustration, but that various modifications may be made withoutdeviating from the spirit and scope of the system. For example, althoughfiles have been described, other types of content such as user settings,application data, emails, and other data objects can be imaged bysnapshots. Accordingly, the system is not limited except as by theappended claims.

Unless the context clearly requires otherwise, throughout thedescription and the claims, the words “comprise,” “comprising,” and thelike are to be construed in an inclusive sense, as opposed to anexclusive or exhaustive sense; that is to say, in the sense of“including, but not limited to.” The word “coupled”, as generally usedherein, refers to two or more elements that may be either directlyconnected, or connected by way of one or more intermediate elements.Additionally, the words “herein,” “above,” “below,” and words of similarimport, when used in this application, shall refer to this applicationas a whole and not to any particular portions of this application. Wherethe context permits, words in the above Detailed Description using thesingular or plural number may also include the plural or singular numberrespectively. The word “or” in reference to a list of two or more items,that word covers all of the following interpretations of the word: anyof the items in the list, all of the items in the list, and anycombination of the items in the list.

The above detailed description of embodiments of the system is notintended to be exhaustive or to limit the system to the precise formdisclosed above. While specific embodiments of, and examples for, thesystem are described above for illustrative purposes, various equivalentmodifications are possible within the scope of the system, as thoseskilled in the relevant art will recognize. For example, while processesor blocks are presented in a given order, alternative embodiments mayperform routines having steps, or employ systems having blocks, in adifferent order, and some processes or blocks may be deleted, moved,added, subdivided, combined, and/or modified. Each of these processes orblocks may be implemented in a variety of different ways. Also, whileprocesses or blocks are at times shown as being performed in series,these processes or blocks may instead be performed in parallel, or maybe performed at different times.

The teachings of the system provided herein can be applied to othersystems, not necessarily the system described above. The elements andacts of the various embodiments described above can be combined toprovide further embodiments.

These and other changes can be made to the system in light of the aboveDetailed Description. While the above description details certainembodiments of the system and describes the best mode contemplated, nomatter how detailed the above appears in text, the system can bepracticed in many ways. Details of the system may vary considerably inimplementation details, while still being encompassed by the systemdisclosed herein. As noted above, particular terminology used whendescribing certain features or aspects of the system should not be takento imply that the terminology is being redefined herein to be restrictedto any specific characteristics, features, or aspects of the system withwhich that terminology is associated. In general, the terms used in thefollowing claims should not be construed to limit the system to thespecific embodiments disclosed in the specification, unless the aboveDetailed Description section explicitly defines such terms. Accordingly,the actual scope of the system encompasses not only the disclosedembodiments, but also all equivalent ways of practicing or implementingthe system under the claims.

While certain aspects of the system are presented below in certain claimforms, the applicant contemplates the various aspects of the system inany number of claim forms. For example, while only one aspect of thesystem is recited as a means-plus-function claim under 35 U.S.C. sec.112, sixth paragraph, other aspects may likewise be embodied as ameans-plus-function claim, or in other forms, such as being embodied ina computer-readable medium. (Any claims intended to be treated under 35U.S.C. § 112, will begin with the words “means for”.) Accordingly, theapplicant reserves the right to add additional claims after filing theapplication to pursue such additional claim forms for other aspects ofthe system.

1. (canceled)
 2. A system comprising: a non-transitory,computer-readable storage medium having computer-executable instructionsstored thereon; and one or more processors that, having executed thecomputer-executable instructions, configure the system to perform aplurality of operations comprising: receiving a first data object to bechunked, wherein: the first data object is stored on a primary storagedevice, and the first data object comprises a first plurality of blocks;chunking the first data object into a first plurality of chunks, whereineach chunk comprises a second plurality of blocks, wherein each secondplurality of blocks is a proper subset of the first plurality of blocks;storing at least some of the first plurality of chunks on a secondarystorage device different from the primary storage device; receiving arequest to modify content of the first data object; identifying one ormore chunks from the first plurality of chunks corresponding to thecontent to be modified; restoring data of the identified one or morechunks of the first plurality of chunks to a client for modifying thecontent associated with the received request; presenting the requestedcontent based on the restored data of the identified one or more chunks;modifying the identified one or more chunks based on modifications tothe presented content; and replacing the identified one or more chunksstored on the secondary storage device with the modified one or morechunks.
 3. The system of claim 1, wherein: the first data objectcomprises a file, and each chunk of the first plurality of chunkscorresponds to a unique portion of the file.
 4. The system of claim 1,wherein: a first chunk of the first plurality of chunks comprisesmetadata of the first data object, the first plurality of chunks otherthan the first chunk comprises payload data of the first data object,and the identifying of the one or more chunks from the first pluralityof chunks corresponding to the content to be modified is based on thefirst chunk of the first plurality of chunks.
 5. The system of claim 1,wherein the plurality of operations further comprises: maintaining adata structure that comprises information about the first plurality ofchunks, wherein the identifying of the one or more chunks from the firstplurality of chunks is based on the data structure; and updating thedata structure based on the modified one or more chunks.
 6. The systemof claim 1, wherein the chunking of the first data object into the firstplurality of chunks is based on at least one of: a minimum data size fora chunk, a maximum data size for a chunk, a time of creation for datawithin a chunk, and a type of data within a chunk.
 7. The system ofclaim 1, wherein the plurality of operations further comprises:identifying a second data object having been previously transferred tothe secondary storage device using data migration other than chunk-baseddata migration; and chunking the second data object into a secondplurality of chunks for future retrieval of content of the second dataobject.
 8. The system of claim 1, wherein content of the first dataobject is assigned to a particular chunk of the first plurality ofchunks based on metadata associated with the content.
 9. A methodcomprising: receiving a first data object to be chunked, wherein: thefirst data object is stored on a primary storage device, and the firstdata object comprises a first plurality of blocks; chunking the firstdata object into a first plurality of chunks, wherein each chunkcomprises a second plurality of blocks, wherein each second plurality ofblocks is a proper subset of the first plurality of blocks; storing atleast some of the first plurality of chunks on a secondary storagedevice different from the primary storage device; receiving a request tomodify content of the first data object; identifying one or more chunksfrom the first plurality of chunks corresponding to the content to bemodified; restoring data of the identified one or more chunks of thefirst plurality of chunks to a client for modifying the contentassociated with the received request; presenting the requested contentbased on the restored data of the identified one or more chunks;modifying the identified one or more chunks based on modifications tothe presented content; and replacing the identified one or more chunksstored on the secondary storage device with the modified one or morechunks.
 10. The method of claim 9, wherein: the first data objectcomprises a file, and each chunk of the first plurality of chunkscorresponds to a unique portion of the file.
 11. The method of claim 9,wherein: a first chunk of the first plurality of chunks comprisesmetadata of the first data object, the first plurality of chunks otherthan the first chunk comprises payload data of the first data object,and the identifying of the one or more chunks from the first pluralityof chunks corresponding to the content to be modified is based on thefirst chunk of the first plurality of chunks.
 12. The method of claim 9,further comprising: maintaining a data structure that comprisesinformation about the first plurality of chunks, wherein the identifyingof the one or more chunks from the first plurality of chunks is based onthe data structure; and updating the data structure based on themodified one or more chunks.
 13. The method of claim 9, wherein thechunking of the first data object into the first plurality of chunks isbased on at least one of: a minimum data size for a chunk, a maximumdata size for a chunk, a time of creation for data within a chunk, and atype of data within a chunk.
 14. The method of claim 9, furthercomprising: identifying a second data object having been previouslytransferred to the secondary storage device using data migration otherthan chunk-based data migration; and chunking the second data objectinto a second plurality of chunks for future retrieval of content of thesecond data object.
 15. The method of claim 9, wherein content of thefirst data object is assigned to a particular chunk of the firstplurality of chunks based on metadata associated with the content.
 16. Anon-transitory, computer-readable storage medium havingcomputer-executable instructions stored thereon that, when executed byone or more processors, configures a system to perform a plurality ofoperations comprising: receiving a first data object to be chunked,wherein: the first data object is stored on a primary storage device,and the first data object comprises a first plurality of blocks;chunking the first data object into a first plurality of chunks, whereineach chunk comprises a second plurality of blocks, wherein each secondplurality of blocks is a proper subset of the first plurality of blocks;storing at least some of the first plurality of chunks on a secondarystorage device different from the primary storage device; receiving arequest to modify content of the first data object; identifying one ormore chunks from the first plurality of chunks corresponding to thecontent to be modified; restoring data of the identified one or morechunks of the first plurality of chunks to a client for modifying thecontent associated with the received request; presenting the requestedcontent based on the restored data of the identified one or more chunks;modifying the identified one or more chunks based on modifications tothe presented content; and replacing the identified one or more chunksstored on the secondary storage device with the modified one or morechunks.
 17. The non-transitory, computer-readable storage medium ofclaim 16, wherein: the first data object comprises a file, and eachchunk of the first plurality of chunks corresponds to a unique portionof the file.
 18. The non-transitory, computer-readable storage medium ofclaim 16, wherein: a first chunk of the first plurality of chunkscomprises metadata of the first data object, the first plurality ofchunks other than the first chunk comprises payload data of the firstdata object, and the identifying of the one or more chunks from thefirst plurality of chunks corresponding to the content to be modified isbased on the first chunk of the first plurality of chunks.
 19. Thenon-transitory, computer-readable storage medium of claim 16, whereinthe plurality of operations further comprises: maintaining a datastructure that comprises information about the first plurality ofchunks, wherein the identifying of the one or more chunks from the firstplurality of chunks is based on the data structure; and updating thedata structure based on the modified one or more chunks.
 20. Thenon-transitory, computer-readable storage medium of claim 16, whereinthe chunking of the first data object into the first plurality of chunksis based on at least one of: a minimum data size for a chunk, a maximumdata size for a chunk, a time of creation for data within a chunk, and atype of data within a chunk.
 21. The non-transitory, computer-readablestorage medium of claim 16, wherein the plurality of operations furthercomprises: identifying a second data object having been previouslytransferred to the secondary storage device using data migration otherthan chunk-based data migration; and chunking the second data objectinto a second plurality of chunks for future retrieval of content of thesecond data object.